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Executive  Summary 


Two  major  purposes  are  served  by  this  report.  The  first  is  to  describe  the 
Synthesized  and  Human  Aerospace  Forces  in  an  Immersive  Research  Environment 
(SAFIRE),  a  simulation  capability  linking  many  of  the  Warfighter  Interface  Division’s 
(AFRL/HEC)  human-in-the-loop  simulations  together  as  well  as  with  external  AFRL  and 
AFMC  assets.  The  second  purpose  is  to  document  the  initial  use  of  the  SAFIRE 
architecture  in  its  intended  role,  as  a  tool  supporting  evaluations  of  crew-system 
interfaces  used  in  a  network-centric  environment. 

Information  availability  was  manipulated  experimentally  during  simulated  air-to- 
air  combat  simulations  of  an  air  base  defense  mission  scenario  that  involved  multiple 
friendly  and  adversary  aircraft  as  well  as  simulated  airborne  command  and  control. 
Statistical  analysis  of  resultant  data  indicated  that  the  manipulation  of  information 
availability  did  affect  both  objective  measures  of  performance  and  subjective  measures 
of  situation  awareness.  The  presence  of  these  effects  clearly  demonstrates  SAFIRE’s 
ability  to  support  crew-system  interface  evaluations.  In  addition,  the  presence  of  a 
practice  effect  was  observed  indicating  more  attention  should  be  focused  on 
experimental  design  issues  for  future  studies  to  reduce  this  effect  or  to  compensate  for 
its  presence. 


The  extensive  implications  of  network-centric  operations  on  the  design  and  use 
of  crew  interfaces  for  current  and  future  Air  Force  systems  is  documented  in  the 
literature.  The  SAFIRE  is  a  powerful  tool  to  investigate  these  implications  in  highly 
complex  and  dynamic  system-of-systems  contexts. 


This  page  intentionally  left  blank. 
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1.  INTRODUCTION 


This  technical  report  has  two  purposes.  First,  to  describe  the  Synthesized  and 
Human  Aerospace  Forces  in  an  Immersive  Research  Environment  (SAFIRE).  Second, 
to  describe  the  initial  network-centric  warfare  experiment  conducted  in  the  SAFIRE. 

The  SAFIRE  has  an  innovative  simulation  capability  for  the  study,  development  and 
evaluation  of  crew-system  interfaces  for  network-centric  operations  and  its  initial  use  for 
this  purpose  will  be  described  in  this  report. 

1.1  Network-Centric  Warfare  Background 

The  United  States  Department  of  Defense  (DoD)  has  indicated  that  the  military 
must  transform  itself  to  more  effectively  deal  with  current  threats.  Two  examples  of  this 
indication  can  be  found  throughout  Joint  Vision  2010  (JV2010,  1996)  and  Joint  Vision 
2020  (JV2020,  2000)  as  the  need  to  move  away  from  a  platform-centric  force  to  a 
network-centric  force. 

One  definition  of  network-centric  warfare  (NOW)  is  “an  information  superiority- 
enabled  concept  of  operations  that  generates  increased  combat  power  by  networking 
sensors,  decision  makers,  and  shooters  to  achieve  shared  awareness,  increased  speed 
of  command,  higher  tempo  of  operations,  greater  lethality,  increased  survivability,  and  a 
degree  of  self  synchronization. ’’(Alberts,  2000).  Network-centric  warfare  is  enabled  by 
technologies  that  enhance  the  gathering,  analysis,  fusion,  and  distribution  of 
information.  The  benefits  of  NCW  are  realized  when  the  decision  quality  of  information 
is  improved  in  content,  format,  and  timeliness,  and  can  support  varying  degrees  of 
autonomy  when  necessary.  The  NCW  model  that  Alberts  has  visualized  is  shown  in 
Figure  1 . 
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Figure  1 -Albert’s  Model  of  Network  Centric  Operations 
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The  NCW  concept  of  operation  is  transformational  in  the  sense  if  moves  defense 
forces  from  platform-centric  operations  to  information-centric  operations.  There  are  a 
significant  number  of  human  factors  challenges  inherent  in  its  implementation  due  in 
part,  to  the  increases  of  information  sources  and  sinks  (Mitchell,  2004).  While  the 
increases  in  information  flow  and  availability  in  network  centric  environments  may 
improve  individual  and  shared  situation  awareness,  they  may  also  increase  the  overall 
workload  and  stress  level  of  individual  crew  members.  The  introduction  of  new  NCW 
concepts  into  any  battlespace  should  be  carefully  considered  with  respect  to  the 
Yerkes-Dobson  Law,  a  commonly  referenced  human  characteristic  is  shown  in  Figure  2. 
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Figure  2-Yerkes-Dodson  Law 

The  Yerkes-Dodson  Law  simply  states  that  performance  is  a  non-linear  function 
of  stress  and/or  the  level  of  arousal  (Boff,  1988).  As  can  be  seen  in  Figure  2, 
performance  can  be  negatively  affected  by  a  stress  level  that  is  “too  high”  or  “too  low”. 
Mitchell  et  a\  equates  stress  with  workload  and  depicts  the  Yerkes-Dodson  Law 
describing  performance  as  a  function  of  workload  (Mitchell,  2004). 

1.2  Initial  NCW  Study 

Characteristics  of  network-centric  warfare,  specifically  self-synchronization,  are 
exhibited  by  pilots  when  flying  air-to-air  combat.  Pilots  continuously  make  decisions 
regarding  their  flight  path  through  three  dimensional  airspace,  their  employment  of 
offensive  weapons,  and  their  defensive  tactics.  This  study  measures  human 
performance  in  simulated  air  combat,  a  “network-centric  environment”,  while  attempting 
to  vary  workload  by  experimentally  manipulating  information  availability  to  the  pilot.  In 
this  experiment,  crew  members  flew  simulated  air  base  defense  missions  under  varying 
levels  of  visual  and  auditory  information  availability. 
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Information  availability  was  manipulated  by  modifying  sensor  capability  of  the 
simulated  aircraft  and  by  the  availability,  or  lack  of,  airborne  command  and  control 
communications.  The  study  utilized  several  components  of  the  SAFIRE  and  fully 
exercised  the  architecture’s  functional  capabilities. 

It  was  hypothesized  that  the  performance  of  the  crews  flying  the  air  base  defense 
missions  would  resemble  a  Yerkes-Dodson  Law  curve  as  the  workload,  or  stress  level, 
was  manipulated  by  information  availability.  This  presence  of  this  effect  is  predicated 
on  the  assertion  that  manipulations  employed  in  the  current  study  span  the  sensitive 
region  of  the  Yerkes-Dodson  curve.  If  this  is  not  the  case,  that  in  itself  would  have  been 
interesting  and  would  provide  important  guidance  for  follow-on  studies. 

1.3  SAFIRE 

The  Synthesized  and  Human  Aerospace  Forces  in  an  Immersive  Research 
Environment  (SAFIRE)  is  a  network  of  research  assets  including  human-in-the-loop 
simulators,  constructive  simulations,  battlespace  visualization  tools,  and  data  collection 
and  analysis  capabilities.  The  SAFIRE  is  architected  using  distributed  simulation 
techniques  for  the  purpose  of  developing  and  evaluating  crew  interface  concepts  for 
network  centric  warfare.  The  SAFIRE’s  unique  combination  of  simulation  assets 
enables  it  to  simulate  many  aspects  of  a  joint  warfighting  environment  and  thus 
represents  the  complexities  inherent  in  the  operation  of  today’s  military  systems-of- 
systems. 

The  purpose  of  the  SAFIRE  capability  is  to  support  the  development  and 
evaluation  of  crew  interface  concepts  employed  in  network  centric  warfare.  It  is  easily 
envisioned  that  crew  interfaces,  controls,  displays,  as  well  as  underlying  automation, 
required  to  optimize  human  performance  in  the  information  rich  environment  of  network 
centric  operations  will  provide  affordances  not  previously  required  (Mitchell,  2004).  In 
addition,  truly  innovative  concepts  of  operations  may  be  utilized  in  a  network  centric 
operation  relative  to  today’s  relatively  platform-centric  operations.  The  ability  of 
researchers  to  manipulate  information  availability  and  portrayal  in  the  SAFIRE,  and 
measure  the  resultant  human  performance  effects,  is  the  focus  of  this  particular  study. 
While  the  study  is  not  an  exhaustive  study  of  each  performance  dimension  of  the 
SAFIRE,  it  is  an  end-to-end  test  along  a  single  dimension. 
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2.  SAFIRE  DESCRIPTION 


AFRL/HEC’s  Synthesized  and  Human  Aerospace  Forces  in  an  Immersive 
Research  Environment  (SAFIRE)  will  be  utilized  during  this  evaluation.  The  SAFIRE  is 
a  combination  of  many  interconnected  human-in-the-loop  (HITL)  simulations  and 
several  constructive  simulation  software  systems.  The  combination  of  constructive 
simulation  and  human-in-the-loop  simulators  provides  the  ability  to  simulate  complex 
and  dynamic  battlefield  situations  involving  air,  sea,  and  ground  components  operating 
interactively  in  real-time.  The  full  SAFIRE  is  depicted  graphically  in  Figure  3. 
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Figure  3.  The  SAFIRE  is  a  collection  of  several  facilities  and  assets  in  different  buildings. 
Typically  data  within  an  end  node  is  exchanged  using  the  DIS  protocol,  and  between  nodes  using 
HLA.  The  SAFIRE  can  accommodate  complex  NCW  experiments. 


The  SAFIRE  architecture  was  developed  to  be  modular  and  scalable  and 
enables  each  of  the  network  end  nodes  to  operate  autonomously  if  needed.  Within 
each  end  node,  data  is  communicated  between  simulation  components  via  the 
Distributed  Interactive  Simulation  (DIS)  protocol.  Between  end  nodes,  data  is 
communicated  using  High  Level  Architecture  (HLA)  constructs.  End  nodes  belonging  to 
a  common  federation  exchange  HLA  data  using  standard  multicast  techniques.  The 
modular  nature  of  the  SAFIRE  architecture  is  shown  in  Figure  4. 
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2.1  Facility  Configuration  for  the  Initial  Study 

Subsets  of  SAFIRE  simulation  capability  can  be  used  to  satisfy  the  requirements 
of  a  specific  test  The  SAFIRE  simulation  capability  brought  together  for  this  evaluation 
is  depicted  in  Figure  5  and  includes  a  low  fidelity  controller  station,  two  desktop  air 
combat  flight  simulations,  JSAF  constructive  simulation  systems,  and  data  collection 
systems. 


Radio  Radio 

315.5  MHz  !■  315.5  MHz 


Figure  5.  The  initial  NCW  experiment  used  an  airborne  controller  (in  the  battlespace 
visualization  room),  the  JSAF  constructive  simulation  tool,  and  two  fighter  simulations  (in  the 
DIGEIE  Blue  facility).  Some  data  collection  assets  were  also  employed. 


Table  1  matches  the  required  simulation  functionality  with  the  method  of  generation  for 
this  evaluation. 


Study  Component  Supplied  by: 

Blue  Fighter  Aircraft  HITL  simulation 

Blue  Airborne  Controller  HITL  simulation 

Red  Strike  Aircraft  JSAF  constructive  simulation 

Red  Fighter  Aircraft  JSAF  constructive  simulation 

Data  Collection _ HLA  Results  and  PIS  Logger  Software 

Table  1  -  Mapping  of  functionality  to  requirements 
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2.2  Blue  Team  Aircraft  Simulators 


The  Blue  team  flew  simulated  fighter  aircraft  in  the  Digital  Entities  for  Interface 
Evaluations  -  BLUE  (DIG-EIE-BLUE)  laboratory  (Figure  6).  DIG-EIE-BLUE  is  equipped 
with  6  Windows  2000  PCs.  These  systems  run  the  SAFIRE  Air  Combat  Simulator 
(SACS),  an  in-house  modified  air-to-air  human-in-the-loop  flight  simulation  (Figure  7). 
Radio  communications  are  simulated  using  the  General  Dynamics  ModlOS  Voice 
Communicator  software.  Stick  and  throttle  controls  are  achieved  using  the 
Thrustmaster  F22  PRO,  Advanced  Control  Series  Joystick  and  SunComm  Strike  Fighter 
Series  Throttle. 


Figure  6.  The  air-to-air  combat  simulators  used  for  the  blue  team  are  hosted  on  desktop  PCs. 
The  simulators  communicate  with  each  other  using  the  Distributed  Interactive  Simulation  (DIS) 
protocol. 
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Figure  7.  This  is  a  screen  capture  from  one  of  the  blue  fighter  crew  stations.  The  top  portion  of 
the  display  is  the  out-the-window  scene  with  the  HUD  overlaid.  The  bottom  section  includes  the 
radar  display,  the  tactical  electronic  warning  system  (TEWS),  engine,  landing  gear,  flaps,  and 
other  status  information.  This  screenshot  was  taken  with  the  two  blue  aircraft  in  a  lead-trail 
formation.  The  other  blue  aircraft  can  be  seen  on  the  TEWS  display  as  a  trailing  blue  dot.  The 
red  dot  on  the  TEWS  display  indicates  the  direction  of  emissions  from  enemy  aircraft. 

The  SAFIRE  Air  Combat  Simulator  (SACS)  is  an  enhanced  version  of  the 
desktop  flight  simulator  software  distributed  by  Web  Simulations,  Inc.  The  radar 
simulation  (Figure  8)  was  modified  to  add  roll  and  pitch  stabilization.  Radar  emission 
information  was  constructed  and  broadcast  using  the  Distributed  Interactive  Simulation 
(DIS)  protocol.  The  SACS  air-to-air  missile  capability  was  upgraded  to  include 
performance  similar  to  the  Advanced  Medium  Range  Air-to-Air  Missile  (AMRAAM). 
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Figure  8.  The  two  blue  aircraft  are  in  a  lead-trail  formation  with  a  range  split  of 
approximately  10  nm.  The  blue  lead  has  designated  a  target  and  fired  a  missile  on  it.  The 
target  information  includes  its  heading,  airspeed,  and  aspect  angle.  The  numbers  in  the  lower 
left  are  the  targets  range  and  bearing  from  the  bullseye  and  its  altitude.  The  trailing  aircraft’s 
radar,  which  includes  the  lead  aircraft,  illustrates  the  concept  of  mutual  radar  support. 


The  stick  and  throttle  were  programmed  to  allow  control  over  aerodynamic 
features  such  as  speed  brakes,  flaps,  trim,  and  afterburner.  A  push-to-talk  switch  was 
implemented  to  allow  keying  of  the  microphone  on  the  voice  over  IP  radio  software.  For 
close-in  air  combat  maneuvers,  the  controls  could  be  used  to  fire  guns.  This  caused  a 
lead  computing  optical  sight  system  (LCOSS)  reticule  to  be  activated  on  the  heads-up 
display  (HUD). 

The  ModlOS  Voice  Communicator  software  enables  two  way  radio 
communications  between  the  pilots  and  the  airborne  operator.  This  software  uses  the 
DIS  standard  to  exchange  data  within  a  facility.  When  radio  communications  are 
required  across  facilities,  the  Network  Interface  Unit  (NIU)  software  is  used  to 
implement  the  HLA  multicast  technique. 
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2.3  Blue  Airborne  Controller  crew  station 


The  Airborne  Command  and  Control  Operator,  located  in  the  Battlespace 
Visualization  room  (Figure  9),  support  the  Blue  fighters  using  the  Virtual  Battlespace 
Management  System  (VBMS)  as  a  visualization  tool  (Figure  10).  This  tool  simulates  an 
airborne  command  and  control  display  (such  as  an  AWACS  or  E-2).  The  controller 
communicates  with  the  Blue  Team  pilots  using  the  simulated  radio  described  earlier.  A 
foot  pedal  switch  was  installed  for  the  controller’s  push  to  talk  device. 


Figure  9.  The  battlespace  visualization  room  housed  the  Virtual  Battlespace  Management 
System  (VBMS).  The  Airborne  Controller  used  this  system  to  make  broadcast  calls  and 
tactical  calls  for  the  read  aircraft.  The  broadcast  calls  were  made  relative  to  the  bullseye,  and 
the  tactical  relative  to  the  requesting  blue  aircraft.  The  Controller  used  a  headset  with  boom 
microphone  and  used  a  push-to-talk  switch  located  near  his  foot. 
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Figure  10.  This  is  a  screen  capture  from  the  Virtual  Battlespace  Management  System  (VBMS). 
This  system  was  used  by  the  airborne  controller  to  make  broadcast  calls  and  tactical  calls  to  the 
two  blue  pilots.  In  this  scene,  range  and  bearing  is  being  measured  from  the  lead  blue  aircraft 
and  the  closest  enemy  aircraft. 
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2.4  Red  Team  Aircraft 


The  Red  Team  bombers  and  fighters  were  provided  and  dynamically  controlled 
using  the  Joint  Simulated  Automated  Forces  (JSAF)  constructive  simulation  tool  (Figure 
1 1 ).  This  tool  allows  several  scenarios  to  be  created  in  advance  and  saved  in  files. 

Prior  to  each  experimental  run,  a  particular  scenario  is  loaded  and  the  red  forces  are 
authorized  to  prosecute  the  attack  on  the  air  base.  Due  to  the  complex  nature  of  JSAF, 
the  scenarios  do  no  simply  play  out  exactly  the  same  way  each  time  they  are  run.  The 
JSAF  entities  dynamically  respond  to  the  actions  of  the  blue  pilots.  This  reactive  nature 
allows  for  realistic,  complex,  and  unique  outcomes  each  time  a  scenario  is  executed. 


Figure  11.  The  JSAF  constructive  simulation  tool  is  used  to  create  scenarios  for  the  red  team  and 
interactively  execute  their  mission  against  the  blue  team  The  two  blue  lighters  are  in  a  CAP  one 
mile  apart  with  one  heading  east  and  one  heading  west. 


The  JSAF  computer  also  contains  a  gateway  that  can  perform  translation 
between  DIS  and  HLA  constructs.  This  powerful  feature  means  that  the  JSAF 
constructive  simulation  tool  can  be  used  in  any  node  in  the  SAFIRE.  In  the  current 
study,  the  JSAF  station  was  located  in  the  Battlespace  Visualization  room. 
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3.  STUDY  DESCRIPTION 

3.1  Introduction 

The  Task:  The  two  blue  fighter  pilots  performed  an  airbase  defense  mission.  The 
blue  team’s  primary  objective  was  to  prevent  the  red  attack  aircraft  from  dropping  any 
bombs  on  the  airbase.  Their  secondary  objective  was  to  delay  the  dropping  of  the  first 
bomb  for  as  long  a  possible  to  allow  ground  assets  on  the  runways  to  be  moved.  The 
blue  pilot’s  primary  source  of  information  for  all  missions  was  the  aircraft’s  attack  radar 
system.  The  range  of  the  radar  system  was  experimentally  controlled.  A  second 
source  of  information  regarding  the  activity  of  the  red  aircraft  is  an  airborne  controller. 
The  controller  was  present  during  one  half  of  the  missions. 

Setup:  At  the  beginning  of  each  trial  the  two  blue  fighters  were  in  a  command  air 
patrol  (CAP)  orbit  14  miles  directly  west  of  the  air  base.  A  bullseye  was  positioned  33 
miles  directly  west  of  the  blue  aircraft’s  starting  position.  The  bullseye  position  was 
shared  between  the  two  blue  aircraft  pilots  and  the  airborne  controller.  The  red  aircraft 
would  ingress  from  several  positions  west,  north,  and  south  of  the  bullseye.  In  each 
mission/trial  there  were  six  red  fighters  and  six  red  attack  (bomber)  aircraft.  Each  blue 
aircraft  was  equipped  with  eight  air-to-air  missiles  and  500  rounds  of  cannon  fire 

Experimental  Hypothesis:  The  hypothesis  of  this  study  is  that  crew  performance 
will  be  affected  by  information  availability  regarding  the  location  and  activity  of  the 
adversary  aircraft,  and,  there  will  be  an  associated  workload  effect.  There  were  two 
manipulations  of  Information  availability.  The  first  was  accomplished  by  limiting  the  blue 
aircraft  to  three  fixed  radar  ranges.  The  second  was  the  presence,  or  lack  of, 
communications  with  an  airborne  controller.  It  is  further  hypothesized  that  the  crew 
performance  effect  will  follow  the  Yerkes-Dodson  Law,  provided  the  information 
manipulations  produce  a  workload  effect. 

3.2  Methods:  Participants 

Blue  Team:  Twelve  air  force  lieutenants  (13  male,1  female)  volunteered  for  the 
study.  All  had  normal,  or  corrected  to  normal,  vision.  The  participants  were  grouped 
into  pairs  to  form  six  blue  teams.  Each  participant  completed  a  signed  consent  form. 
The  training  and  data  collection  took  approximately  24  hours  over  a  three  month  period. 

Airborne  Controller:  One  controller  was  utilized  throughout  the  study.  The 
controller  was  a  trained  E-2  controller  with  1 ,000  hours  of  combat  experience.  The  role 
of  the  controller  was  to  provide  information,  relative  to  the  red  force  formation  and 
composition.  The  controller  was  strictly  a  source  of  information  and  was  explicitly  told 
not  to  direct  the  blue  aircraft.  The  controller  was  unaware  of  the  radar  range  settings  for 
the  blue  aircraft.  Generally,  the  controller  would  make  broadcast  calls  (relative  to 
bullseye),  but  when  asked  by  the  pilots  for  snap  vectors,  the  controller  would  make 
tactical  calls  (relative  to  the  aircraft). 
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3.3  Methods:  Experimental  Design 


Independent  Variables: 

Radar  Display  Range  (20,  40,  and  80  nm) 

Airborne  Controller  (Present  vs.  Not  Present) 

Dependent  Variables: 

Number  of  successful  missions 

Bombing  delay 

Number  of  bombers  destroyed 

Number  of  red  fighters  destroyed 

Number  of  blue  fighters  lost 

Workload 

Situational  awareness 

For  a  mission  to  be  successful,  it  meant  that  all  of  the  red  attack  aircraft 
(bombers)  were  destroyed  prior  to  dropping  any  bombs.  It  was  possible  to  kill  all  of  the 
red  bombers  and  not  prevent  bombing  of  the  airbase.  This  happened  on  at  least  one 
trial  where  an  egressing  bomber  was  shot  down  after  releasing  ordinance,  and  then  an 
inbound  bomber  was  destroyed. 

The  bombing  delay  was  an  important  measure.  The  longer  the  bombers  were 
delayed,  the  more  time  ground  troops  would  have  to  move  assets  from  the  runways.  A 
baseline  delay  was  established  by  letting  each  scenario  play  out  without  the  blue  aircraft 
present.  The  bombing  delay  was  measured  as  time  beyond  the  baseline  when  the  first 
bomb  was  released.  The  number  of  successful  missions  and  the  bombing  delay  are 
mutually  exclusive  measures. 

Workload:  The  Subjective  Workload  Assessment  Technique  (SWAT)  was  used 
to  collect  subjective  workload  measures  after  each  trial.  Each  participant  received  the 
SWAT  training  and  performed  the  SWAT  card  sort  prior  to  data  collection. 

Situational  Awareness:  The  Situational  Awareness  Rating  Technique  (SART) 
was  used  to  collect  subjective  situational  awareness  measures  after  each  trial. 

Scenarios:  There  were  three  different  scenarios  created  for  the  red  team  using 
the  JSAF  constructive  simulation  tool.  Each  of  these  three  scenarios  contained  the 
same  number  of  fighter  and  attack  aircraft  with  different  starting  positions.  The  three 
scenarios  were  not  treated  as  an  independent  variable,  but  were  implemented  to 
prevent  the  pilot  from  learning  a  single  scenario  and  thus  predicting  behavior. 

The  study  utilized  a  within  subject  (team)  design.  The  Presence  of  Controller 
variable  was  blocked.  The  order  of  Radar  Range,  Presence  of  Controller,  and  scenario 
were  counterbalanced 
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3.4  Procedures:  Trial  Format 


Prior  to  each  data  collection  session,  a  checklist  was  followed  to  ensure  that  the 
pilot  stations  were  calibrated  correctly  and  that  the  radio  communications  were  working 
properly.  Each  trial  also  had  a  set  of  procedures  to  follow  to  ensure  that  all  tasks  were 
accomplished  in  the  correct  order.  The  DIS  Logger  was  started  prior  to  each  trial. 

To  begin  the  trial,  the  blue  teams’  fighter  aircraft  would  be  activated  flying  in  the 
CAP  with  the  predetermined  radar  range.  At  this  time,  the  appropriate  red  team 
scenario  would  be  started  in  the  Battlespace  Visualization  Laboratory.  If  the  block  of 
trials  included  the  airborne  controller,  he  would  use  the  Virtual  Battlespace  Management 
System  (VBMS)  and  begin  making  “calls”  when  he  saw  that  the  red  aircraft  were 
present. 

The  two  blue  team  pilots  were  instructed  to  remain  in  the  CAP  until  they  had  a 
reason  to  leave  it  (i.e.,  they  had  an  awareness  of  the  red  aircraft).  This  could  happen  in 
one  of  three  ways.  First,  if  there  was  an  airborne  controller  he  would  alert  the  blue  team 
when  the  red  aircraft  appeared  on  his  screen.  Second,  one  or  more  red  aircraft  would 
become  visible  on  the  blue  team’s  radar,  and  third,  one  of  the  blue  aircraft  would  get  a 
spike  on  their  defensive  display  from  emissions  by  one  of  the  Red  aircraft.  At  this  point 
the  blue  pilots  would  break  out  of  the  CAP  and  engage  the  adversaries. 

The  blue  pilots  were  free  to  use  tactics  of  their  choice  to  perform  the  mission. 

The  pilots  were  not  told  to  use  a  specific  tactic,  nor  were  they  directed  to  do  so  by  the 
airborne  controller.  During  the  Advanced  Tactics  section  of  training  regimen,  the  pilots 
were  given  the  opportunity  to  practice  several  well  defined  tactics. 

Each  trial  lasted  approximately  ten  minutes.  A  trial  was  discontinued  when  one 
of  the  following  criteria  was  met: 

1 )  The  mission  was  successful  (all  bombers  were  destroyed  before  dropping 
ordinance). 

2)  The  first  bomb  had  dropped  on  the  runway  and  the  blue  team  was  no  longer 
able  to  affect  the  outcome  of  the  remaining  bombers  (e.g.,  both  blue  were 
dead  or  out  of  weapons). 

3)  All  bombers  had  released  their  ordinance  and  were  egressing. 

A  “knock  if  off”  call  was  made  when  one  of  the  above  criteria  were  met.  Mission 
outcome  was  scored  manually  during  the  trial  and  then  verified  in  an  after-action-review 
using  the  DIS  Logs.  Subjective  measures  were  collected  following  each  trial.  The 
subjects  were  not  given  any  specific  feedback  on  their  performance  after  the  trial. 
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3.5  Procedures:  Training 

All  of  the  eligible  candidates  who  volunteered  for  the  study  had  to  progress 
through  a  highly  structured  training  procedure.  Participants  where  not  allowed  to 
advance  to  the  next  stage  until  they  established  competency  in  the  current  stage. 

Basic  Flying  Skills:  The  basic  flying  skills  portion  of  the  training  involved 
familiarization  with  the  controls  and  displays  of  the  simulation.  Each  participant  was 
asked  to  fly  at  various  altitudes,  air  speeds  and  headings;  and  to  trim  the  aircraft  at  each 
condition. 

Basic  Targeting  Skills:  Once  the  participant  displayed  competency  in  flying  the 
aircraft,  then  the  concepts  of  the  bullseye,  radar  operation,  target  designation  and 
weapon  delivery  were  introduced.  This  included  recognizing  changes  to  the  radar 
symbology  and  the  appearance  of  the  target  designator  (TD)  box  on  the  HUD.  To 
practice  the  switchology,  the  participants  used  a  simple  scenario  with  stationary  targets. 

Advanced  Targeting  Skills:  The  participants  were  introduced  to  the 
communications  system  (simulated  radios  with  push-to-talk).  The  participants  took 
turns  at  being  the  lead  aircraft  and  being  the  wingman.  When  in  the  lead  role,  that 
participant  would  assign  targets  to  the  wingman  using  calls  relative  to  the  bullseye.  The 
participants  flew  against  a  simple  scenario  where  the  red  aircraft  were  live  and  had  the 
ability  to  fire  on  the  blue  aircraft.  In  this  phase  of  training  the  tactical  electronic  warning 
system  (TEWS)  was  explained. 

Communications:  The  subjects  watched  a  video  on  Brevity  given  by  an 
experienced  controller.  The  AFTTP  3-1,  Volume  1,  Change  1, 2  Dec  1999 
ATTACHMENT  1-1  OPERATIONAL  BREVITY  WORDS,  TERMINOLOGY,  and  AIR-TO- 
AIR  COMMUNICATION  STANDARDS  (U)  was  used  as  guidelines  for  the  terminology 
Upon  completion  of  the  video,  the  participants  were  introduced  to  the  Airborne 
Controller  and  flew  against  the  Advance  Targeting  scenario  while  communicating  with 
the  controller.  The  correct  use  of  terminology  was  reinforced  throughout  this  phase. 

Tactics:  The  participants  watched  a  video  presented  by  a  retired  fighter  pilot 
(Shaw  1985)  who  addressed  some  basic  tactics  and  defensive  maneuvers.  The  video 
addressed  strategies  to  maximize  weapon  effectiveness  and  how  to  defeat  the 
opponents’  weapons.  After  the  video,  the  participants  were  briefed  on  the  details  of  the 
airbase  defense  mission.  They  were  also  introduced  to  the  CAP  (Combat  Air  Patrol) 
concept.  Following  the  discussion,  the  participants  were  given  the  opportunity  to 
practice  the  tactics  discussed  in  the  video. 

Advanced  Tactics:  The  participants  watched  a  brief  video  presented  by  an 
experienced  fighter  pilot  who  discussed  maneuvering  terminology  and  team  oriented 
tactics.  The  participants  then  flew  against  practice  scenarios  and  were  given  interactive 
tactical  instruction.  During  this  instruction  the  participants  were  given  a  specific  tactic  to 
try.  The  participants  were  then  debriefed  to  discuss  the  pros  and  cons  of  the  specific 
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tactic.  In  this  session  the  participants  completed  the  Subjective  Workload  Assessment 
Technique  (SWAT)  card  sort.  The  specific  tactics  were: 

1 )  Guard  home  plate  and  the  bullseye:  The  two  blue  aircraft  would  split  up 
with  one  going  back  to  the  airbase  and  circling  the  runway  waiting  for  red 
aircraft  to  come  into  its  radar  range.  The  other  blue  aircraft  would  fly  west  to 
the  bullseye  and  maintain  a  CAP-like  orbit  until  it  could  engage  the  bandits. 
This  is  a  conservative  tactic  that  teaches  patience. 

2)  Engage  and  guard  home  plate:  Similar  to  the  above  except  the  second  blue 
aircraft  doesn’t  stop  at  the  bullseye.  It  keeps  flying  west  searching  for  the 
adversary.  This  tactic  illustrates  the  dangers  of  engaging  the  enemies  as  a 
single  ship  while  demonstrating  the  importance  of  preventing  “leakers”  from 
slipping  through  to  bomb  the  base. 

3)  Both  Engage:  With  this  tactic  both  blue  aircraft  immediately  fly  west  in  an 
abreast  formation  to  search  out  and  engage  the  enemy  as  far  from  the 
airbase  as  possible.  This  tactic  allows  the  participants  to  practice  evasive 
maneuvers  and  the  highlights  possible  advantages  of  being  very  aggressive. 

4)  CAP-like  shoot-and-turn:  Both  blue  aircraft  begin  flying  west  with  a  range 
split  of  approximately  15  nm.  When  the  lead  aircraft  is  within  weapon  range, 
he  shoots  and  turns  east,  eventually  passing  by  the  other  blue  aircraft,  which 
then  becomes  the  lead.  This  aircraft  then  shoots  and  turns,  and  the  pattern 
repeats.  This  tactic  illustrates  the  concept  of  mutual  radar  support. 

5)  North-South  azimuth  split.  The  two  blue  aircraft  split  with  one  flying 
southwest  and  one  flying  northwest  until  they  are  30-40  nm  apart.  They  then 
engage  separate  enemies  that  they  encounter.  This  tactic  avoids  the  pitfalls 
of  the  fly  straight  in  and  engages  tactics,  which  can  lead  to  a  red  team  trap 
(i.e. ,  ambush).  Additionally  it  illustrates  the  advantage  of  spreading  out  the 
radar  coverage  to  prevent  red  attack  aircraft  from  slipping  around  the  flanks 
and  attacking  the  airbase. 

6)  High-Low  elevation  split:  One  blue  aircraft  would  fly  directly  west  at  low 
altitude  (1  OK  feet)  and  the  other  would  fly  west  while  climbing  to  a  high 
altitude  (40K  feet)  and  substantially  increase  airspeed.  This  tactic 
emphasizes  the  advantage  of  having  a  high-energy  weapon  release 
condition.  It  also  illustrates  the  method  to  increase  radar  coverage  in  the 
vertical  dimension. 

3.6  Procedures:  Team  Formation  and  Practice 

Based  on  availability  and  progress  through  the  training  regimen,  participants 
were  paired  to  form  a  team.  Once  a  team  was  established,  they  were  given  the 
opportunity  to  practice  with  and  without  the  controller.  During  these  practice  sessions 
the  necessity  of  tactics  was  continually  reinforced.  The  participants  were  not  limited  to 
the  tactics  taught  in  the  Advanced  Tactics  phase  of  training.  They  could  use  a 
combination  of  those  tactics  or  develop  and  practice  new  ones.  However,  it  was 
strongly  emphasized  that  when  a  trial  begins,  the  team  must  have  decided  on  a  tactic 
and  it  must  be  clearly  communicated.  The  team  remained  in  the  practice  phase  of 
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training  until  they  achieved  50%  success  against  the  practice  scenarios.  At  that  point 
they  proceeded  to  the  Mission  Rehearsal  phase. 


3.7  Procedures:  Mission  Rehearsal 

During  mission  rehearsal  the  participants  flew  against  even  harder  scenarios  with 
all  the  conditions  (20,  40,  80  mile  radar  range,  with  and  without  a  controller).  Subjective 
workload  and  situational  awareness  measures  (SART  and  SWAT)  were  taken  after 
each  trial.  The  participants  were  allowed  to  ask  questions  during  mission  rehearsal,  but 
minimal  guidance  was  provided. 

3.8  Procedures:  Data  Collection 

Prior  to  data  collection  the  participants  were  given  a  written  set  of  instructions 
(Appendix  B)  to  read  and  any  remaining  questions  were  answered.  During  the  data 
collection  trials  no  guidance  was  provided  by  the  experimenters.  The  participants  had 
to  rely  on  their  training,  communication  with  each  other,  and  communication  with  the 
Airborne  Controller,  if  present. 

Three  new  red  team  scenarios  (Appendix  A)  were  created  using  the  JSAF 
constructive  simulation  tool.  These  scenarios  were  more  difficult  that  the  ones  used  in 
the  mission  rehearsal  stage  of  training.  Data  collection  consisted  of  a  total  of  18  trials 
(three  scenarios,  three  radar  ranges,  and  Airborne  Controller  present  or  absent). 
Typically  the  18  trials  were  completed  in  two  sessions  on  two  separate  days.  However, 
this  was  not  always  possible  do  to  the  military  commitments  of  the  participants. 
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4.  RESULTS 


Data  representing  8  dependent  variables  (6  objective  and  2  subjective)  were 
collected  from  7  teams  of  participants.  The  data  were  collected  during  trials  in  which 
two  independent  variables  were  systematically  manipulated.  Data  were  averaged 
across  the  three  scenarios  for  each  team  and  treatment  combination. 

4.1  Definition  of  Variables 

Names  of  the  dependent  and  independent  variables,  along  with  their  associated 
descriptions,  are  shown  in  Table  2  below. 


Variable  Name 

Variable  Description 

Type 

Success 

Percentage  of  successful 
missions(  no  bombs  detonated 
on  airbase) 

Dependent 

Delay 

Difference  between  baseline 
time  of  scenario  and  time  from 
scenario  start  to  first  bomb 
detonation  on  airbase 

Dependent 

Bombers 

Number  of  adversary  bombers 
shot  down  in  a  single  mission 

Dependent 

Escorts 

Number  of  adversary  fighters 
shot  down  in  a  single  mission 

Dependent 

Bandits 

Number  of  adversary 
aircraft(fighters  and  bombers) 
shot  down  in  a  single  mission 

Dependent 

Fighters 

Number  of  friendly  fighters  shot 
down 

Dependent 

Workload 

Subjective  workload  measured 
using  the  SWAT  methodology 

Dependent 

SA 

Situation  Awareness  measured 
using  the  SART  methodology 

Dependent 

Range 

Range  of  radar  display  in  friendly 
fighter  simulations 

Independent 

Controller 

Availability  of  friendly  airborne 
controller 

Independent 

4.2  Primary  Graphical  Results 

Data  collected  during  this  study  representing  the  dependent  variables  as  a 
function  of  the  independent  variables  is  depicted  graphically  in  Figures  12  through  19 
below. 
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Figure  12  -  Percentage  of  Successful  Missions. 
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Figure  13  -  Average  Bombing  Delay  per  Mission. 
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Figure  14  -  Average  Number  of  Bandits  Shot  Down  per  Mission. 
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Figure  15  -  Average  Number  of  Bombers  Shot  Down  per  Mission. 
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Figure  16  -  Average  Number  of  Escorts  Shot  Down  per  Mission. 
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Figure  17  -  Average  Number  of  Friendly  Fighters  Fost  per  Mission. 
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Figure  18  -  Average  Team  Subjective  Workload  per  Mission. 
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Figure  19  -  Average  Team  Situation  Awareness  per  Mission. 
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4.3  Primary  ANOVA  Results 

Although  many  of  these  figures  resemble  the  Yerkes-Dodson  curve,  and  thus 
support  the  experimental  hypothesis,  a  statistical  analysis  of  the  data  must  be 
completed  before  reliable  conclusions  can  be  drawn. 

Eight  two-way  analyses  of  variance  were  performed  on  the  collected  data.  The 
analyses  of  variance  indicated  that  the  two  independent  variables  interacted  to 
significantly  affect  the  number  of  bandits  shot  down  (F=6.24(2,12),(p  <  .05),  the  number 
of  escorts  shot  down  (F  =  10.58(2,12),  p  <  .05)  and  Situation  Awareness  (F  = 

9.56(2, 1 2),  p  <  .05).  The  results  of  eight  two-way  analyses  of  variance  of  the  objective 
and  subjective  dependent  variables  are  shown  below.  It  should  be  noted  that  the 
missing  data  in  the  analysis  of  Delay  was  attributable  to  all  missions  in  that  particular 
cell  were  successful  (no  bombs  were  dropped  on  the  airbase  and  thus  the  variable 
Delay  was  indeterminate. 


ANALYSIS  OF  VARIANCE  TABLE  FOR  SUCCESS 


SOURCE  DF  SS  MS  F  P 


TEAM  (A) 

e 

0 . 48148 

0 . 08025 

CONTROLLER  (B) 

i 

0 . 06614 

0 . 06614 

1.23 

0.3100 

A*B 

e 

0.32275 

0 . 05379 

RANGE  (C) 

2 

0 . 19048 

0 . 09524 

1 . 48 

0.2665 

A*C 

12 

0.77249 

0.06437 

B*C 

2 

0.08466 

0.04233 

0.87 

0 . 4427 

A*B*C 

12 

0.58201 

0 . 04850 

TOTAL 

41 

2.50000 

ANALYSIS  OF 

VARIANCE 

TABLE  FOR 

DELAY 

SOURCE 

DF 

SS 

MS 

F 

P 

TEAM  (A) 

6 

20.5299 

3.42164 

2.04 

0.2028 

CONTROLLER 

(B)  1 

1 . 96401 

1 . 96401 

1 . 17 

0.3203 

A*B 

6 

10.0428 

1 . 67380 

RANGE  (C) 

2 

0 . 97699 

0 . 48849 

0.21 

0 . 8107 

A*C 

12 

27 . 4512 

2.28760 

B*C 

2 

0.88566 

0 . 44283 

0.20 

0 . 8193 

A*B*C 

11 

24 . 0049 

2 . 18227 

TOTAL 

40 

85.8555 

CASES  INCLUDED  41  MISSING  CASES  1 

CAUTION:  The  sums  of  squares,  mean  squares,  and  F-tests  are  approximate 

for  analyses  with  missing  values.  See  the  manual  for  details  and 
instructions  for  constructing  exact  F-tests. 
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ANALYSIS  OF  VARIANCE  TABLE  FOR  BANDITS 


SOURCE 

DF 

SS 

MS 

F 

P 

TEAM  (A) 

e 

27 . 4527 

4.57545 

RANGE  (B) 

2 

0.89616 

0 . 44808 

0.16 

0.8543 

A*B 

12 

33.6941 

2 . 80784 

CONTROLLER  (C) 

1 

4 . 45399 

4 . 45399 

0.64 

0 . 4540 

A*C 

e 

41 . 7198 

6.95329 

B*C 

2 

22 . 7589 

11.3795 

6.24 

0.0139 

A*B*C 

12 

21 . 8989 

1 . 82491 

TOTAL 

41 

152 . 875 

ANALYSIS  OF  VARIANCE  TABLE  FOR  BOMBERS 


SOURCE  DF  SS  MS  F  P 


TEAM  (A) 

6 

15.1480 

2.52467 

CONTROLLER  (B) 

1 

2 . 43372 

2 . 43372 

1 . 09 

0.3367 

A*B 

6 

13.3954 

2.23256 

RANGE  (C) 

2 

2 . 17650 

1.08825 

1 . 17 

0.3426 

A*C 

12 

11.1353 

0 . 92794 

B*C 

2 

4 . 64578 

2.32289 

1.77 

0.2126 

A*B*C 

12 

15.7801 

1.31501 

TOTAL 

41 

64 . 7148 

ANALYSIS  OF 

VARIANCE 

TABLE  FOR 

ESCORTS 

SOURCE 

DF 

SS 

MS 

F 

P 

TEAM  (A) 

6 

4.03238 

0.67206 

CONTROLLER 

(B)  1 

0.30295 

0.30295 

0 . 14 

0.7213 

A*B 

6 

12 . 9952 

2 . 16587 

RANGE  (C) 

2 

1 . 97945 

0 . 98972 

0.73 

0.5010 

A*C 

12 

16.2112 

1.35093 

B*C 

2 

6.85856 

3.42928 

10.58 

0.0022 

A*B*C 

12 

3 . 89074 

0.32423 

TOTAL  41  46.2705 


ANALYSIS  OF  VARIANCE  TABLE  FOR  FIGHTERS 


SOURCE  DF  SS  MS  F  P 


TEAM  (A) 

6 

2 . 07702 

0.34617 

CONTROLLER  (B) 

1 

0 . 02445 

0 . 02445 

0.04 

0 . 8407 

A*B 

6 

3.32744 

0.55457 

RANGE  (C) 

2 

0.01562 

0 . 00781 

0.05 

0.9531 

A*C 

12 

1 . 94383 

0 . 16199 

B*C 

2 

0.39557 

0 . 19778 

0.86 

0 . 4461 

A*B*C 

12 

2 . 74728 

0.22894 

TOTAL  41  10.5312 
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ANALYSIS  OF  VARIANCE  TABLE  FOR  WORKLOAD 


SOURCE  DF  SS  MS  F  P 


TEAM  (A) 

6 

2158 . 87 

359.812 

CONTROLLER  (B) 

1 

886.249 

886.249 

2 . 94 

0 . 1371 

A*B 

6 

1807 . 46 

301.244 

RANGE  (C) 

2 

401.545 

200.773 

2.07 

0 . 1689 

A*C 

12 

1163.72 

96.9767 

B*C 

2 

120 . 455 

60.2275 

0.27 

0.7706 

A*B*C 

12 

2714.23 

226.186 

TOTAL  41  9252.54 


ANALYSIS  OF  VARIANCE  TABLE  FOR  SA 


SOURCE  DF  SS  MS  F  P 


TEAM  (A) 

6 

9.19108 

1.53185 

CONTROLLER  (B) 

1 

1.77988 

1 . 77988 

2.39 

0 . 1729 

A*B 

6 

4 . 46471 

0.74412 

RANGE  (C) 

2 

2.21284 

1 . 10642 

2 . 19 

0.1551 

A*C 

12 

6.07437 

0.50620 

B*C 

2 

2.30067 

1 . 15033 

9.56 

0.0033 

A*B*C 

12 

1 . 44448 

0 . 12037 

TOTAL  41  27.4680 


4.4  Secondary  Analyses:  Effects  of  Stage 

A  two-factor  within-subjects  design  was  utilized  in  this  study  with  resulting  data 
being  averaged  across  three  scenarios  for  each  combination  of  team  and  treatment 
condition.  While  not  a  repeated  measures  design  by  rigorous  definition,  data  resulting 
from  the  utilization  of  the  three  scenarios  could  be  viewed  as  three  repeated  measures. 
Additionally,  there  were  several  tactics  that  may  have  been  employed  by  the 
participants  while  attempting  to  defend  the  airbase  and  those  tactics  may  have  been 
changing  throughout  the  data  collection  period.  Many  of  these  tactics  were  presented 
to,  and  practiced  by,  the  participants  during  their  training.  The  selection  of  tactics  was 
not  constrained  during  data  collection.  The  scenarios  utilized  during  data  collection 
were  different  than  those  used  during  training.  The  combination  of  these  elements 
makes  it  essential  to  analyze  the  data  for  the  potential  existence  of  “practice  effects”, 
effects  embedded  in  the  data  resulting  from  the  adaptation  of  the  participants  to  the 
experimental  conditions. 

Shaughnessy  and  Zechmeister(Shaughnessy,  1990)  recommend  separate 
statistical  analyses  for  practice  effects  and  for  independent  variable  effects.  A  variable 
must  be  introduced  to  perform  the  separate  analysis  representing  the  sequential  order 
of  the  independent  variable  presentations.  Although  arbitrary,  the  most  straight-forward 
approach  to  create  this  new  variable  from  this  dataset  was  to  break  the  data  from  each 
team  into  the  two  blocks  associated  with  the  presence  or  absence  of  the  controller. 
Stage  was  introduced  into  the  dataset  as  the  new  variable  with  a  value  of  1  representing 
the  first  9  trials  of  each  team  and  the  value  2  representing  the  last  9  trials  from  each 
team.  Of  the  7  teams,  3  teams  had  the  controller  present  in  Stage  1  and  4  teams  had 
the  controller  present  in  Stage  2.  To  statistically  evaluate  if  “practice  effects”  may  have 
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been  present  in  the  dataset,  two  dependent  variables  previously  determined  not  to  have 
been  significantly  affected  by  the  independent  variables  were  analyzed  using  State  as  a 
within-subject  independent  variable.  These  analyses  revealed  that  both  the  number  of 
bombers  shot  down  (Bombers)  and  subjective  workload  (Workload),  were  significantly 
affected  by  Stage  (F  =  4.58(1,6),  p<  .05  and  F  =  6.08(1,6),  p  <  .05  respectively). 
Graphical  depictions,  as  well  as  results  of  the  analyses  of  variance  are  shown  below. 


42  cases 

Figure  20  -  Number  of  Bombers  Shot  Down  as  a  Function  of  Stage, 
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Figure  21  -  Subjective  Workload  as  a  Function  of  Stage. 


ANALYSIS  OF 

VARIANCE 

TABLE  FOR 

BOMBERS 

SOURCE 

DF 

ss 

MS 

F 

P 

TEAM  (A) 

6 

15 . 1480 

2.52467 

STAGE  (B) 

1 

5.51770 

5.51770 

4.58 

0.0412 

A*B 

6 

10.3114 

1 . 71856 

1.43 

0.2398 

RESIDUAL 

28 

33.7377 

1.20492 

TOTAL 

41 

64 . 7148 

ANALYSIS  OF 

VARIANCE 

TABLE  FOR 

WORKLOAD 

SOURCE 

DF 

ss 

MS 

F 

P 

TEAM  (A) 

6 

2158 . 87 

359.812 

STAGE  (B) 

1 

955.897 

955.897 

6.08 

0.0200 

A*B 

6 

1737 . 82 

289.636 

1 . 84 

0.1267 

RESIDUAL 

28 

4399.95 

157 . 141 

TOTAL 

41 

9252.54 
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4.5  Secondary  Analyses:  Graphs  and  ANOVAs 

The  clear  statistical  indications  of  “practice  effects”  in  the  dataset  compel  an 
additional  analysis  of  the  data  from  only  Stage  2.  However,  this  secondary  analysis 
using  only  Stage  2  data  must  be  performed  with  the  independent  variable  Controller 
being  treated  as  a  between-subject  manipulation  and  the  independent  variable  Range 
treated  as  a  within-subject  manipulation.  In  essence,  the  dataset  must  be  analyzed  as 
a  mixed  two  factor  design  and  is  complicated  by  differing  numbers  of  teams  in  the  two 
levels  of  the  Controller  variable. 

Graphs  of  the  eight  dependent  variables  are  shown  below. 
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Figure  22  -  Percentage  of  Successful  Missions. 


29 


BANDITS  DELAY(seconds) 


CONTTROL 


RANGE(nm) 

20  cases  1  missing  cases 

Figure  23  -  Average  Bombing  Delay  per  Mission. 
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Figure  24  -  Average  Number  of  Bandits  Shot  Down  per  Mission. 
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Figure  25  -  Average  Number  of  Bombers  Shot  Down  per  Mission. 


CONfTROL 

NO 

YES 


RANGE(nm) 

21  ca*s 

Figure  26  -  Average  Number  of  Escorts  Shot  Down  per  Mission. 


31 


WORKLOAD(%)  FIGHTERS 


2- 


1- 


0- 


Figure 


CONTTROL 
*  NO 
□  YES 


20 


40 


80 


RANGE(nm) 

21  cases 

27  -  Average  Number  of  Friendly  Fighters  Lost  per  Mission. 
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Figure  28  -  Average  Team  Subjective  Workload  per  Mission. 


32 


6- 


CONTTROL 
*  NO 
□  YES 


1- 


0- 

20  40  80 


RANGE(nm) 

21  cases 

Figure  29  -  Average  Team  Situation  Awareness  per  Mission. 


The  results  of  the  secondary  statistical  analyses  indicated  that  SA  was 
significantly  affected  by  the  interaction  of  radar  range  (Radar)  and  the  presence  of  the 
controller(Controller)  (F=4. 607(2, 10),  p  <  .05).  The  results  of  the  eight  secondary 
statistical  analyses  are  shown  below. 


Dep  Var:  SUCCESS  N:  21 

Multiple 

R:  1 

.000  Squared  multiple  R: 

1.000 

Analysis  of 

Variance 

Source  Sum- 

■of-Squares 

df 

Mean-Square 

F-ratio 

P 

RANGE 

2222 . 095 

2 

1111 . 048 

1.250 

0.328 

CONTROLLER$ 

1428 . 524 

1 

1428 . 524 

1.286 

0.308 

CONTROLLER?  *RANGE 

952.381 

2 

476.190 

0.536 

0.601 

TEAM (CONTROLLER?) 

5555.889 

5 

1111 . 178 

RANGE* TEAM (CONTROLLER?) 

8889.333 

10 

888.933 

Error 

0.000 

0 

Dep  Var:  DELAY  N:  21  Note:  1  Missing  value 

Analysis  of  Variance 


Source 

Sum-of -Squares 

df 

Mean-Square 

F-ratio 

P 

RANGE 

2.252 

1 

2.252 

0.958 

0.351 

CONTROLLER? 

2 . 871 

1 

2 . 871 

1.227 

0.318 

CONTROLLER?  *RANGE 

1.706 

2 

0.853 

0.363 

0 . 704 

TEAM (CONTROLLER?) 

11 . 702 

5 

2.340 

RANGE* TEAM (CONTROLLER?)  23.494 

10 

2.349 

Error 

0.000 

0 
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Dep  Var:  BOMBERS 

N:  21 

Multiple 

R: 

1.000 

Squared  multiple  R: 

1.000 

Analysis 

of 

Variance 

Source 

Sum- of- 

■Squares 

df 

Mean- 

■Square 

F-ratio 

P 

RANGE 

1.593 

2 

0 . 797 

0.933 

0.425 

CONTROLLER? 

2.912 

1 

2 . 912 

2.021 

0.214 

CONTROLLER?  *RANGE 

4.587 

2 

2.294 

2.686 

0.116 

TEAM (CONTROLLER? ) 

7.206 

5 

1 . 441 

RANGE  *  TEAM ( CONTROLLER? ) 

8.539 

10 

0 . 854 

Error 

0.000 

0 

Dep  Var :  ESCORTS 

N:  21 

Multiple 

R: 

1.000  Squared  multiple  R: 

1.000 

Analysis 

of  Variance 

Source 

Sum- of- 

■Squares 

df 

Mean-Square 

F-ratio 

P 

RANGE 

1 . 793 

2 

0.896 

1.016 

0.397 

CONTROLLER? 

1.361 

1 

1.361 

1.355 

0.297 

CONTROLLER?  *RANGE 

1.993 

2 

0.996 

1.129 

0.361 

TEAM (CONTROLLER? ) 

5.023 

5 

1.005 

RANGE  *  TEAM ( CONTROLLER? ) 

8.822 

10 

0.882 

Error 

0.000 

0 

Dep  Var:  FIGHTERS 

N:  21 

Multiple 

R: 

1.000  Squared  multiple  R: 

1.000 

Analysis  of  Variance 

Source 

Sum- of- 

■Squares 

df 

Mean-Square 

F-ratio 

P 

RANGE 

0.124 

2 

0.062 

0.301 

0.747 

CONTROLLER? 

0.162 

1 

0.162 

0.311 

0.601 

CONTROLLER?  *RANGE 

0 . 498 

2 

0.249 

1.212 

0.338 

TEAM (CONTROLLER? ) 

2.606 

5 

0.521 

RANGE*TEAM (CONTROLLER?) 

2 . 057 

10 

0.206 

Error 

0.000 

0 

Dep  Var:  BANDITS  N:  21 

Multiple 

R: 

1.000  Squared  multiple  R: 

1.000 

Analysis 

of  Variance 

Source  Sum- 

of-Squares 

df 

Mean-Square 

F-ratio 

P 

RANGE 

6.509 

2 

3.254 

1 . 156 

0.354 

CONTROLLER? 

8.277 

1 

8.277 

2.502 

0 . 175 

CONTROLLER?  *RANGE 

10.008 

2 

5.004 

1 . 777 

0.219 

TEAM (CONTROLLER? ) 

16.540 

5 

3.308 

RANGE*TEAM (CONTROLLER?) 

28 . 155 

10 

2.816 

Error 

0.000 

0 
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Dep  Var:  WORKLOAD 

N:  21  Multiple 

R: 

1.000  Squared  multiple  R: 

1.000 

Analysis  of  Variance 

Source 

Sum-of -Squares 

df 

Mean-Square 

F-ratio 

P 

RANGE 

159.491 

2 

79.746 

0.490 

0.627 

CONTROLLER? 

71 . 990 

1 

71 . 990 

0 . 137 

0.727 

CONTROLLER?  *RANGE 

120.730 

2 

60.365 

0.371 

0.699 

TEAM (CONTROLLER? ) 

2629.756 

5 

525.951 

RANGE* TEAM (CONTROLLER?)  1628.006 

10 

162.801 

Error 

0.000 

0 

Dep  Var:  SA  N:  21  Multiple  R:  1.000  Squared  multiple  R:  1.000 

Analysis  of  Variance 


Source  Sum-of- 

■Squares 

df 

Mean-Square 

F-ratio 

P 

RANGE 

0.699 

2 

0.350 

1 . 105 

0.368 

CONTROLLER? 

1.332 

1 

1.332 

0.675 

0 . 449 

CONTROLLER?  *RANGE 

2.914 

2 

1 . 457 

4 . 607 

0.038 

TEAM (CONTROLLER? ) 

9.865 

5 

1 . 973 

RANGE  *  TEAM ( CONTROLLER? ) 

3.163 

10 

0.316 

Error 

0.000 

0 

5.  DISCUSSION  AND  CONCLUSIONS 

Albert’s  model  of  network  centric  operations  portrays  a  sequence  of  effects 
beginning  with  information  availability  affecting  situation  awareness  which  in  turn  affects 
task  performance.  The  statistically  significant  interactions  found  in  this  study  of  radar 
range  and  the  presence  of  the  controller  on  the  situation  awareness  of  the  simulation 
pilots  and  on  the  number  of  bandits  shot  down  supports  the  constructs  of  this  model.  In 
addition,  the  reduction  in  situation  awareness  in  the  20  nm  range  without  the  controller 
relative  to  the  other  radar  ranges,  partially  supports  the  presence  of  a  “Yerkes-Dodson 
curve”  to  the  shape  of  the  data.  The  lack  of  statistically  significant  reductions  at  the  80 
nm  range  and  the  lack  of  increases  in  situation  awareness,  workload,  and  performance 
when  the  controller  was  present  do  not  support  the  presence  of  a  “Yerkes-Dodson 
curve”  shape  to  the  data. 

The  statistically  significant  difference  in  performance  and  workload  observed 
between  the  first  and  last  halves  of  trials  was  quite  unexpected.  It  is  not  clear  from  the 
data  the  cause  of  these  shifts.  One  plausible  explanation  may  have  been  that  the 
participants  became  increasingly  familiar  with  the  actions  of  the  adversary  aircraft  and 
their  pre-planned  flight  paths.  Alternatively,  the  shifts  may  have  been  due  to 
optimization  of  tactics  utilized.  Because  no  constraints  were  imposed  on  the 
participants  in  terms  of  choice  of  tactics,  the  same  three  scenarios  being  used  the  same 
number  of  times  in  both  trial  periods,  and  there  typically  being  a  significant  time  gap 
between  the  first  and  last  half  of  data  collection  periods,  the  participants  may  have 
chosen  what  they  considered  to  be  “the  best”  tactic  and  repeatedly  used  it  during  the 
second  half  of  the  trials.  This  issue  should  be  addressed  in  future  studies  to  determine 
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the  cause  of  this  shift  and,  in  turn,  to  better  control  its  effects  in  future  studies  having 
similar  characteristics  to  this  study. 

The  secondary  analysis  which  involved  analyzing  only  data  from  the  second  half 
of  the  trials  revealed  a  similar  interaction  on  situation  awareness  but  the  significant 
interaction  observed  in  performance  was  no  longer  present.  It  is  difficult  to  draw  reliable 
conclusions  from  these  data  in  that  for  each  combination  of  independent  variable  level, 
only  3  or  4  data  points  are  present  and  that  the  unbalanced  nature  of  the  data  requires 
the  analysis  of  variance  to  be  estimated.  With  this  in  mind,  it  could  be  speculated  that 
the  reduction  in  power  of  the  secondary  analyses  had  masked  observable  effects.  This 
speculation  is  somewhat  substantiated  by  inspection  of  the  graphical 
portrayal  of  the  data  and  the  similar  nature  of  the  trends  in  the  means. 

One  overall  conclusion  to  be  drawn  from  the  description  of  the  SAFIRE  and  the 
results  of  this  initial  study  is  that  the  SAFIRE  is  capable  of  hosting  evaluations  of  NOW 
interface  concepts  and  that  there  is  a  need  to  further  understand  the  implications  of 
information  characteristics  in  the  design  of  crew  interfaces  to  be  used  in  network-centric 
operation. 


36 


6.  REFERENCES 


Taylor,  R.M.  (1990).  Situational  awareness  rating  technique  (SART):  The  development 
of  a  tool  for  aircrew  systems  design.  In  Situational  Awareness  in  Aerospace  Operations 
(AGARD-CP-478,  pp.  3-1  and  3-17).  Neuilly  Sur  Seine,  France:  NATO-AGARD. 

Alberts,  D.S.,  Garstka,  J.J.,  and  Stein,  F.P.  (2000).  Network  Centric  Warfare: 
Developing  and  Leveraging  Information  Superiority,  Command  and  Control  Research 
Program  (CCRP),  Washington  D.C. 

Mitchell,  P.J.,  Cummings,  M.L.m  and  Sheridan,  T.B.  ((2004).  Human  Supervisory 
Control  Issues  in  Network  Centric  Warfare,  HAL2004-01,  Massachusetts  Institute  of 
Technology,  Department  of  Aeronautics  and  Astronautics,  Cambridge,  MA. 

Reid,  G.B.,  Potter,  S.S.,  and  Bressler,  J.R.  (1989)  Subjective  Workload  Assessment 
Technique  (SWAT):  A  User’s  Guide,  AAMRL-TR-89-023,  Harry  G.  Armstrong 
Aerospace  Medical  Research  Laboratory,  Wright-Patterson  Air  Force  Base,  OH. 

Boff,  K.R.  and  Lincoln,  J.E.  (1988)  Engineering  Data  Compendium,  Human  Perception 
and  Performance,  Harry  G.  Armstrong  Aerospace  Medical  Research  Laboratory, 
Wright-Patterson  Air  Force  Base,  OH. 

Shaughnessy,  J.  J.  and  Zechmeister,  E.  B..  (1990)  Research  Methods  in  Psychology, 
(2nd.  Ed.).  New  York \  McGraw-Hill 

JV2020  (2000)  Joint  Vision  2020,  Available  at:  http://www.dtic.mil/jointvision/jvpub2.htm 
Director  for  Strategic  Plans  and  Policy,  J5;  Strategy  Division,  US  Government  Printing 
Office,  Washington  DC 

JV2010  (1996)  Joint  Vision  2010,  Available  at:  http://www.dtic.mil/jointvision/jvpub.htm 
Director  for  Strategic  Plans  and  Policy,  J5;  Strategy  Division,  US  Government  Printing 
Office,  Washington  DC 

Shaw,  R.L.,  (1985),  FIGHTER  COMBAT  Tactics  and  Maneuvering,  Naval  Institute 
Press,  Annapolis,  MD. 


37 


APPENDIX  A:  RED  TEAM  SCENARIOS 


The  three  scenario  used  for  data  collection  had  6  fighter  aircraft  and  6  strike  aircraft 
(bombers).  The  fighters  displayed  very  aggressive  behavior  while  the  bombers  were 
more  passive  and  generally  only  engaged  when  on  the  defensive.  The  red  fighters 
were  escorts  for  the  strikers.  Their  primary  mission  was  to  destroy  the  blue  teams’ 
aircraft  to  allow  the  strike  aircraft  to  complete  their  mission  of  bombing  the  airbase. 

Scenario  One:  This  scenario  begins  with  four  fighters  (2  groups  of  2),  at  medium 
altitude,  30  miles  west  of  the  bullseye,  two  fighters  at  high  altitude  13  miles  south- 
southwest  of  the  bullseye,  four  bombers,  at  medium  altitude,  40  miles  west  of  the 
bullseye  (10  miles  in  trail  of  the  lead  division  of  fighters),  and  two  bombers,  at  high 
altitude,  50  miles  northwest  of  the  bullseye. 

When  the  trial  begins,  the  southern  fighters  fly  a  sweep  from  south  to  north  and  then 
head  east  toward  the  airbase;  the  northwest  bombers  fly  a  northeast  to  southeast  arc  to 
the  airbase;  the  lead  fighter  division  split  into  two  sections,  one  heading  northeast  then 
east,  the  other  southeast  then  east;  the  western  bomber  division  flies  east,  then 
retrogrades  west,  then  turns  back  east  to  the  airbase. 


Scenario  1.  Screen  shot  of  the  JSAF  constructive  simulation  tool  showing  the  layout  of  scenario 
one.  The  two  blue  fighters  are  in  a  CAP  one  mile  apart  with  one  heading  east  and  one  heading 
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Scenario  Two:  This  scenario  begins  with  four  fighters  (two  groups  of  two)  at  low 
altitude,  31  miles  west-northwest  of  the  bullseye.  The  northern  group  is  at  low  altitude 
and  southern  group  is  at  high  altitude.  The  other  two  fighters  are  at  medium  altitude, 
seven  miles  in  trail  of  the  northern  group.  Two  bombers  begin  45  miles  northwest  of  the 
bullseye,  at  high  altitude.  The  other  four  bombers  begin  at  low  altitude,  in  a  wall 
formation  10  miles  west  of  the  fighter  sections,  stacked  slightly  northwest  to  southeast 

When  the  trial  begins,  the  low  altitude  fighters  flow  northwest  and  then  east;  the  high 
fighters  flow  directly  east;  the  northwest  bombers  drive  southeast  to  the  airbase;  the 
fighters  from  the  wall  flow  southwest  and  then  east;  the  bombers  flow  southeast, 
retrograde  back  west  and  then  flow  back  to  the  east  to  the  airbase 


V  -  -  X 


|ij|  |  Zoom:  click  middle  to  zoom  in  around  point;  click  right  to  zoom  out  around  point;  click  and  drag  middle  to  set  screen  area  L/L  — i  |  Jb5"46'35.3Z"N  |[l  1 6“1 3‘22.80"W 

|  Select  an  item  to  edit  j 

Scenario  2.  Screen  shot  of  the  JSAF  constructive  simulation  tool  showing  the  layout  of  scenario 
one.  The  two  blue  fighters  are  in  a  CAP  one  mile  apart  with  one  heading  east  and  one  heading 
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Scenario  Three:  This  scenario  begins  with  two  fighters  47  miles  north  of  bullseye,  at 
high  altitude,  two  fighters  28  miles  southwest  of  the  bullseye,  at  medium  altitude,  and 
two  fighters  30  miles  west  of  the  bullseye,  at  high  altitude.  Four  of  the  bombers  start  40 
miles  west  of  the  bullseye,  stacked  medium  to  low.  The  other  two  bombers  start  45 
miles  west  of  the  bullseye,  stacked  high  to  medium 

When  the  trial  begins,  the  northern  fighters  flow  south  and  then  east;  the  southern 
fighters  flow  northwest  then  east;  the  lead  fighters  in  the  ladder  do  a  cross-over  and  flow 
northeast  and  then  east,  and  southeast  and  then  east;  the  division  of  bombers  split  to 
northeast  to  east  and  southeast  to  east;  and  the  section  of  bombers  flow  southeast  to 
east  to  the  airbase. 


Scenario  3.  Screen  shot  of  the  JSAF  constructive  simulation  tool  showing  the  layout  of  scenario 
one.  The  two  blue  fighters  are  in  a  CAP  one  mile  apart  with  one  heading  east  and  one  heading 
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APPENDIX  B:  INSTRUCTIONS  GIVEN  TO  PARTICIPANTS 


Purpose:  The  study  you  are  participating  in  supports  an  AFRL/HEC  research  initiative 
to  define  and  overcome  the  human  factors  issues  associated  with  the  implementation  of 
the  Network  Centric  Warfare  concept  of  operations.  Secretary  of  Defense  Donald 
Rumsfield  has  stated  that  the  DoD  must  transform  itself  from  a  cold  war  capability  to 
one  appropriate  for  the  present  threats  (Transformation  Planning  Guidance,  Department 
of  Defense,  2003).  Alberts  states  that  Network  Centric  Warfare  is  an  information 
superiority-enabled  concept  of  operation  that  generates  increased  combat  power  by 
networking  sensors,  decision-makers,  and  shooters  to  achieve  shared  awareness, 
increased  survivability,  and  a  degree  of  self-synchronization  (Network  Centric  Warfare, 
Alberts,  Gartska  and  Stein,  1999).  The  Network  Centric  Warfare  concept  of  operation  is 
transformation  and  there  are  a  significant  number  of  human  factors  challenges  inherent 
in  its  implementation.  Exploratory  development  in  this  area  enables  AFRL/HEC  to 
develop  interface  concepts  that  will  optimize  the  capability  of  warfighters  operating  in 
future  Network  Centric  environments. 

Mission  Description:  You  will  be  flying  an  air-to-air  combat  air  base  defense  mission 
through  hostile  air  space.  You  will  have  one  wingman.  You  and  your  wingman  are  the 
“DIGEIE  team”.  Your  task  is  to  locate  and  destroy  the  computer  controlled  enemy 
bombers  (whose  mission  is  to  bomb  Nellis  AFB  that  you  are  protecting).  The  scenario 
will  also  include  independently  manned  hostile  fighters  whose  task  is  to  eliminate  the 
DIGEIE  team  and  defend  the  bombers.  It  is  possible  that  other  blue  team  aircraft  might 
be  returning  from  another  mission.  Those  aircraft  will  be  coming  from  a  vector  of  225 
from  Nellis  AFB  (homeplate). 

Each  mission  will  begin  with  the  DIGEIE  team  flying  in  a  Combat  Air  Patrol  (CAP) 
pattern  until  the  hostile  aircraft  are  detected.  The  mission  will  be  terminated  when  the 
DIGEIE  team  aircraft  have  successfully  destroyed  all  the  bombers  OR  all  the  bombers 
have  crossed  homeplate  (Nellis  AFB). 

Rules  of  Engagement:  You  may  employ  any  tactics  necessary  to  destroy  the  bombers 
flying  in  toward  Nellis,  including  destroying  the  enemy  fighters.  The  threats  may 
approach  from  the  North,  South  or  West  of  Nellis,  AFB. 

Additional  information: 

DIGEIE  team  radar  range  will  vary  by  trial. 

DIGEIE  team  may  or  may  not  have  the  assistance  of  a  controller. 

Blue  team  missile  range  30  mi. 

Red  missile  range  approximately  21  miles. 

One  DIGEIE  team  member  will  start  at  90  from  the  bullseye  and  the  other  will  start  at 
270  from  the  bullseye. 

Nellis  AFB  is  approximately  30  miles  East  of  the  bullseye. 

Following  each  trial  you  will  be  asked  for  some  subjective  workload  (SWAT)  and 
situational  awareness  (SART)  ratings. 
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APPENDIX  C:  INFORMED  CONSENT  FORM 


INFORMATION  PROTECTED  BY  THE  PRIVACY  ACT  OF  1974 

Informed  Consent  Document 
For 

Network-Centric  Interface  Development  and  Evaluation  using  SAFIRE 
AFRL/HECP,  WPAFB,  OH,  Building  33 


Principal  Investigator:  Michael  W.  Haas,  Principal  Electronics  Engineer,  DSN  785-8768, 

AFRL/HECP,  michael.haas@wpafb.af.mil 

Associate  Investigators:  Shari  Ulring,  Human  Factors  Engineer,  DSN  785-5 1 12, 

AFRL/HECP ,  shari . ulring@wpafb .  af . mil 

1 .  Nature  and  purpose:  You  have  been  offered  the  opportunity  to  participate  in  the  “SAFIRE 
Human-centered  Simulation  Architecture  Evaluation”  research  study.  Your  participation  will 
occur  sometime  between  15  December  2005  and  14  December  2006,  in  the  Synthesized  and 
human  Aerospace  Forces  in  an  Immersive  Research  Environment  (SAFIRE)  at  Wright- 
Patterson  AFB.  The  SAFIRE  is  a  set  of  distributed  simulation  facilities  digitally  linked 
together  using  distributed  simulation  technologies. 

The  purpose  of  this  research  is  to  determine  if  the  SAFIRE  simulation  can  be  used  to  develop 
and  evaluate  crew  systems  for  future  Air  Force  aircraft  and  ground  control  stations. 

The  data  collection  time  requirement  for  each  volunteer  subject  is  anticipated  to  be  a  total  of 
2  visits  of  approximately  3-4  hours.  Training  for  each  subject  will  be  tailored  to  individual 
needs  and  progress  on  a  self-paced  schedule.  The  maximum  training  time  will  be 
approximately  16  hours  spread  across  several  days.  A  total  of  approximately  20  subjects  will 
be  enrolled  in  this  evaluation  in  pairs  of  two.  You  will  be  required  to  report  to  the 
experimenter  that  you  possess  nonnal  or  corrected  to  normal  visual  acuity  and  nonnal 
hearing  to  be  eligible  for  participation  in  this  study. 

2.  Experimental  procedures:  If  you  decide  to  participate,  you  will  be  asked  to  sit  at  a  desktop 
fighter  simulator  and  perfonn  air-to-air  combat  tasks.  You  will  be  asked  to  complete  surveys 
to  provide  your  feedback  on  the  interface  technologies  and  displays  you  experience.  Testing 
will  last  3-4  hours  per  day  and  will  be  done  in  normal  lighting  conditions.  You  will  be  seated 
during  the  data  collection  trials.  You  will  be  offered  a  rest  period  midway  through  the  testing 
period  but  can  request  a  break  at  any  time  during  the  experiment.  You  may  withdraw  this 
consent  at  any  time  and  discontinue  further  participation  in  this  evaluation  without  prejudice 
to  your  entitlements.  Also  understand  that  the  medical  monitor  of  this  evaluation  may 
tenninate  your  participation  in  this  evaluation  if  she  or  he  feels  this  to  be  in  your  best  interest. 
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3.  Discomfort  and  risks:  There  is  minimal  risk  and/or  discomfort  associated  with  performing 
this  task.  Mild  postural  fatigue  and  eye  strain  has  been  shown  to  be  prevalent  in  continued 
computer  usage  but  is  normally  alleviated  with  rest  breaks. 

4.  Precautions  for  female  subjects:  There  are  no  special  precautions  for  female  subjects. 

5.  Benefits:  For  active  duty  participants,  there  are  no  additional  direct  benefits  to  you  for 
participation.  Subject  pool  participants  will  be  compensated  monetarily  at  the  prevailing 
General  Dynamics  AIS  rate. 


6.  Alternatives:  Choosing  not  to  participate  is  an  alternative  to  volunteering  for  this 

evaluation. 

7.  Entitlements  and  confidentiality: 

a.  Records  of  your  participation  in  this  evaluation  may  only  be  disclosed  according  to 
federal  law,  including  the  Federal  Privacy  Act,  5  U.S.C.  552a,  and  its  implementing 
regulations. 

b.  You  understand  your  entitlements  to  medical  and  dental  care  and/or  compensation  in  the 
event  of  injury  are  governed  by  federal  laws  and  regulations,  and  that  if  you  desire  further 
information  you  may  contact  the  base  legal  office  (88  ABW/JA,  257-6142  for  Wright- 
Patterson  AFB).  You  may  contact  the  medical  monitor,  Dr.  Jeff  Bidinger,  Maj.,  USAF, 
of  this  research  evaluation  at  (937)  255-4563. 

c.  If  an  unanticipated  event  (medical  misadventure)  occurs  during  your  participation  in  this 
evaluation,  you  will  be  informed.  If  you  are  not  competent  at  the  time  to  understand  the 
nature  of  the  event,  such  infonnation  will  be  brought  to  the  attention  of  your  next  of  kin. 

Next  of  Kin  if  needed,  Name _ ,  Phone# _ . 

d.  The  decision  to  participate  in  this  research  is  completely  voluntary  on  your  part.  No  one 
has  coerced  or  intimidated  you  into  participating  in  this  program.  You  are  participating 
because  you  want  to.  Michael  Haas,  or  an  associate,  has  adequately  answered  any  and  all 
questions  you  have  about  this  evaluation,  your  participation,  and  the  procedures  involved. 
Michael  Haas  can  be  reached  at  (937)  255-8768.  You  understand  that  Michael  Haas,  or 
an  associate  will  be  available  to  answer  any  questions  concerning  procedures  throughout 
this  evaluation.  You  understand  that  if  significant  new  findings  develop  during  the 
course  of  this  research,  which  may  relate  to  your  decision  to  continue  participation,  you 
will  be  informed.  You  further  understand  that  you  may  withdraw  this  consent  at  any  time 
and  discontinue  further  participation  in  this  evaluation  without  prejudice  to  your 
entitlements.  You  also  understand  that  the  medical  monitor  of  this  evaluation  may 
tenninate  your  participation  in  this  evaluation  if  she  or  he  feels  this  to  be  in  your  best 
interest.  If  you  have  any  questions  or  concerns  about  your  participation  in  this  evaluation 
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or  your  rights  as  a  research  subject,  please  contact  Major  Jeff  Bidinger  at  (937)  255-4563 
or  ieffrey.bidinger@wpafb.af.mil. 

e.  You  understand  that  your  participation  in  this  evaluation  may  be  photographed,  filmed  or 
audio/videotaped.  The  audio/video  data  will  be  used  for  data  analysis,  data  retrieval  and 
backup  purposes  only.  All  audio/video  media  will  be  stored  in  a  secure  cabinet  for  up  to 
5  years  in  the  BMC2  Lab,  Bldg  33,  WPAFB,  OH.  You  understand  that  any  release  of 
records  of  your  participation  in  this  evaluation  may  only  be  disclosed  according  to  federal 
law,  including  the  Federal  Privacy  Act,  55  U.S.C.  552a,  and  its  implementing  regulations. 
This  means  personal  infonnation  will  not  be  released  to  unauthorized  sources  without 
your  permission. 

f.  YOU  FULLY  UNDERSTAND  THAT  YOU  ARE  MAKING  A  DECISION  WHETHER 
OR  NOT  TO  PARTICIPATE.  YOUR  SIGNATURE  INDICATES  THAT  YOU  HAVE 
DECIDED  TO  PARTICIPATE  HAVING  READ  THE  INFORMATION  PROVIDED 
ABOVE. 

Volunteer  Signature _ Date _ 

Volunteer  Social  Security  No.  (Optional) _ 

Advising  Investigator  Signature _ Date _ 

Witness  Signature _ Date _ 


Privacy  Act  Statement 


Authority:  We  are  requesting  disclosure  of  personal  information,  to  include  your  Social 
Security  Number.  Researchers  are  authorized  to  collect  personal  information  (including  social 
security  numbers)  on  research  subjects  under  The  Privacy  Act-5  USC  552a,  10  USC  55,  10 
USC  8013,  32  CFR  219,  45  CFR  Part  46,  and  EO  9397,  November  1943. 

Purpose:  It  is  possible  that  latent  risks  or  injuries  inherent  in  this  experiment  will  not  be 
discovered  until  some  time  in  the  future.  The  purpose  of  collecting  this  information  is  to  aid 
researchers  in  locating  you  at  a  future  date  if  further  disclosures  are  appropriate. 

Routine  Uses:  Information  (including  name  and  SSN)  may  be  furnished  to  Federal,  State  and 
local  agencies  for  any  uses  published  by  the  Air  Force  in  the  Federal  Register,  52  FR  16431,  to 
include,  furtherance  of  the  research  involved  with  this  evaluation  and  to  provide  medical  care. 
Disclosure:  Disclosure  of  the  requested  information  is  voluntary.  No  adverse  action 
whatsoever  will  be  taken  against  you,  and  no  privilege  will  be  denied  you  based  on  the  fact  you 
do  not  disclose  this  information.  However,  your  participation  in  this  evaluation  may  be  impacted 
by  a  refusal  to  provide  this  infonnation. 
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APPENDIX  D:  SACS  OPERATIONS  MANUAL 


Introduction 


The  purpose  of  this  document  is  to  familiarize  pilots  and  other  users  with  the  SAFIRE  Air 
Combat  Simulator  (SACS)  prior  to  participation  in  any  simulation.  This  documentation  should 
be  used  in  conjunction  with  a  formal  hands-on  training  session  with  the  simulator  controls  and 
displays. 

SACS  Display 

The  SACS  display  is  comprised  of  two  sections.  The  top  section  is  the  out-the-window  scene 
with  the  Heads-up  Display  (HUD)  overlaid.  The  bottom  section  contains  instrumentation,  status 
information,  the  radar  display,  and  the  tactical  electronic  warfare  system  (TEWS).  The  SACS 
display  is  shown  below  in  Figure  1 . 
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THE  HEADS  UP  DISPLAY  (HUD) 

The  HUD  contains  primary  flight  information  that  permits  pilots  to  focus  their  attention  on 
what's  going  on  outside  of  the  cockpit  without  looking  down  at  the  instruments.  The  HUD 
symbology  is  overlaid  on  the  out-the-window  scene  and  shown  in  Figure  2. 


G-meter 


Heading  (306  degrees)  Angle  of  attack 


Weapon  state 

(8  x  AIM-9M  Sidewinders  Armed 


Altitude 
(1200  ft 


Figure  2  -  SACS  HUD  Symbology 


Airspeed  Tape-  On  the  left  side  of  the  display  is  the  plane's  true  airspeed.  Each  major  tick  mark 
is  50  knots  and  each  minor  tick  mark  is  10  knots.  The  airspeed  tape  slides  up  and  down  and  is 
read  against  the  reference  line  in  the  middle. 

Altitude  Tape  -  The  altitude  tape,  displayed  on  the  right  side,  is  similar  to  the  airspeed  tape 
except  each  major  tick  mark  represents  500  feet  and  each  minor  tick  mark  is  100  feet.  The 
altitude  is  defined  as  feet  above  mean  sea  level  (MSL).  .  A  vertical  velocity  readout,  expressed 
in  feet  per  minute,  is  located  just  below  the  altitude  tape. 

Heading  Tape  -  In  the  upper  center,  displayed  horizontally,  is  the  heading  tape.  Each  tick 
represents  five  degrees.  The  accompanying  numerics  are  tens  of  degrees  (i.e.,  31  equals  a 
heading  of  310). 

Flight  Path  Marker  -  The  flight  path  marker  is  displayed  in  the  center  of  the  HUD  as  a  circle 
with  three  short  lines  attached  to  it.  The  flight  path  marker  indicates  the  instantaneous  velocity 
vector  of  the  aircraft  (i.e.,  its  direction  of  travel  at  any  point  in  time). 

Pitch  Ladder  -Centered  about  the  flight  path  marker  is  the  pitch  ladder.  Each  line  corresponds  to 
ten  degrees  of  aircraft  pitch  (solid  lines  for  positive  pitch  and  dashed  lines  for  negative  pitch). 

The  pitch  angle  is  measured  against  the  pitch  reference  symbol  (the  small  circle  in  the  middle). 
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Angle  of  Attack  -  The  plane’s  current  angle  of  attack,  expressed  in  degrees,  is  located  above  the 
altitude  tape  following  the  a=  symbols.  The  AC  will  stall  at  a  positive  angle  of  30  degrees  and  a 
negative  angle  of  -30  degrees. 

G-Meter  -  A  readout  of  the  current  vertical  G-force  on  the  pilot  is  located  on  the  left-hand  side, 
above  the  airspeed  tape. 

Weapons  -  The  currently  selected  weapon  status,  quantity,  and  type  is  displayed  below  the 
airspeed  tape.  If  a  target  has  been  designated  by  the  pilot,  a  target  designator  box  (a  box  slightly 
larger  than  the  flight  path  marker)  will  be  displayed  at  the  location  of  the  target,  and  the  range  to 
the  target  will  follow  the  weapon  status  readout. 


The  Radar  Display 

The  radar  system  has  a  field  of  view  that  extends  +/-  60  degrees  laterally  and  +/-  20  degrees 
vertically  from  the  front  of  the  aircraft  (Figure  2).  The  radar  coverage  is  roll  and  pitch  stabilized. 
The  radar  display  range  is  shown  in  the  upper  right  hand  corner  of  the  radar  display.  All  aircraft 
within  that  range  and  field  of  view  will  appear  as  small  unfilled  squares  on  the  radar  display.  In 
addition  each  aircraft  depicted  on  the  display  will  have  a  line  extending  from  the  square 
indicating  the  relative  direction  of  travel 

The  vertical  dimension  of  the  radar  display  is  the  distance  to  the  target.  The  horizontal 
dimension  is  the  angle-off-nose. 

Using  a  button  on  the  throttle,  the  pilot  can  cycle  through  the  displayed  targets  to  choose  one  of 
them  to  be  the  designated  target.  The  symbol  for  the  designated  target  will  change  from  a  small 
unfilled  square  to  a  filled  square.  When  a  target  is  designated,  additional  information  is 
displayed,  including  the  target  aspect  angle,  range  to  the  target,  and  closure  rate. 


Blue  Lead  Aircraft’s  Radar 
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Tactical  Electronic  Warfare  System  (TEWS) 

The  TEWS  display  is  located  just  to  the  left  of  the  radar  display.  It  is  the  two  concentric  circles 
with  the  ownship  symbol  in  the  center.  When  radar  emissions  from  other  aircraft  are  detected, 
the  system  plots  a  box  on  the  TEWS  display  in  the  relative  direction  of  the  radar  emission.  If  the 
detected  emission  is  beyond  20  nautical  miles,  the  box  is  plotted  on  the  perimeter  of  the  display. 
As  the  emitting  aircraft  comes  closer  that  20  mn,  the  box  moves  inward  toward  the  ownship 
symbol.  The  box  will  be  colored  red  if  the  emissions  are  from  an  enemy  aircraft  and  blue  for 
friendly  aircraft.  This  receiver  cannot  detect  aircraft  that  have  their  radar  emissions  directed 
away  from  your  aircraft,  nor  is  it  capable  of  detecting  aircraft  that  have  their  radar  sets  turned 
completely  off.  An  additional  component  of  the  warning  system  is  an  audio  warning  that  gets 
played  when  an  air-to-air  missile  has  acquired  a  radar  lock  on  your  aircraft. 

Instrumentation 

Additional  elements  of  the  lower  portion  of  the  SACS  display  include  landing  gear  status, 
throttle  setting,  engine  RPM,  current  position,  flaps  setting,  and  subsystems  failure  indicators. 

Weapon  Systems 

The  aircraft  is  equipped  with  medium  range  missiles  and  guns/cannon.  Weapon  information  is 
displayed  in  the  lower  left-hand  corner  of  your  HUD.  The  desired  weapon  is  selected  by  using 
the  thumb  switch  on  the  stick.  The  currently  selected  weapon  is  fired  by  pulling  the  trigger 
switch  on  the  stick 

Air-to-Air  Missiles 

The  missiles  are  patterned  after  U.S.  AIM- 120  AMRAAM.  These  are  fire-and-forget  missiles 
with  a  maximum  range  of  approximately  30  mn.  Their  range  varies  dramatically  with  the 
altitude  of  both  aircraft  and  their  closure  rate.  The  missile  subsystem  couples  with  the  radar 
system  to  attain  targeting  parameters.  The  missile  launch  status  displayed  on  the  HUD  has  three 
states,  armed,  locked,  and  no-escape.  In  this  simulation  the  missiles  are  always  anned.  They 
transition  to  "locked"  when  the  target  was  within  range  and  in  the  radar  field  of  view.  They 
transition  to  no-escape  when  there  is  a  high  probability  of  kill  based  on  range  and  closure  rate. 
The  no-escape  state  was  achieved  normally  at  a  range  of  12-15  mn. 

Cannon 

Cannon/guns  are  used  to  engage  targets  at  closer  range  (i.e.,  less  than  1.5  mn).  The  cannon  is 
patterned  after  the  U.S.  M61-A1  Vulcan.  Missions  begin  with  500  rounds  of  ammunition 

When  the  cannon  is  selected,  a  Lead  Computing  Optical  Sight  (LCOS)  reticule  is  displayed  on 
the  HUD.  The  HUD  couples  with  the  radar  to  provide  a  visible  cue  of  the  target's  current  range. 
The  aiming  reticule  is  surrounded  by  12  ticks.  An  inner  arc  represents  the  current  range  to  the 
target:  each  arc  tick  represents  1000  feet  of  distance.  The  aiming  reticule  moves  across  the  HUD 
to  show  a  good  aiming  point  based  on  the  target's  range  and  your  aircraft's  pitch  and  turn  rate.  If 
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the  range  is  large  and  or  your  pitch  and  turn  rates  are  fast,  you  may  see  no  reticule  at  all:  the 
aiming  point  is  simply  out  of  the  HUD's  field  of  view. 

A  fixed  cross  (a  "+")  is  displayed  on  the  HUD  to  indicate  the  bore  sight  of  the  cannon  —  the 
direction  that  the  cannon’s  barrel  is  actually  pointing.  A  good  shot  is  possible  when  the  target  is 
within  range  and  the  LCOS  reticule  is  centered  on  the  bore  site  cross. 

The  HSI 

The  Horizontal  Situation  Indicator  (HSI)  is  a  device  to  aid  navigation.  The  HSI  shares  space 
with  the  radar  display.  The  desired  display  can  be  selected  using  a  button  on  the  stick.  The  HIS 
can  be  used  to  put  the  aircraft  on  a  heading  towards  a  selected  airbase. 


Selected  Course  Pointer 


Glide  Slope  Deviation  Indicator 


Glide  Slope  Scale 


Course  Deviation  Indicator 


NAVAID  Frequency 


Figure  3  -  SACS  HSI 


SACS  Controls 

The  SACS  aircraft  can  be  piloted  using  a  mouse  and  keyboard,  or  by  attaching  a  stick  and 
throttle  quadrant  to  the  computer.  The  list  of  keyboard  commands  can  be  displayed  by  hitting 
the  FI  key  when  SACS  is  running.  When  using  an  attached  stick  and  throttle,  various  buttons 
and  switches  are  mapped  to  the  desired  keystrokes 

Hands-on  Throttle  and  Stick  (HOT AS)  Controls 

The  HOTAS  control  configurations  are  shown  in  Figure  4  and  Figure  5.  The  following 
paragraphs  describe  the  functionality  assigned  to  the  buttons  and  switches. 


Point-of-View  -  The  out-the-window  scene  can  be  generated  by  looking  out  the  front  of  the 
aircraft,  to  the  left  or  right,  or  out  the  back  of  the  aircraft.  The  view  can  be  changed  using  the  top 
center  hat  (looks  like  a  castle)  on  the  stick. 

Trim  -  The  aircraft  aerodynamics  can  be  trimmed  using  the  top  right  hat  on  the  stick.  When  the 
aircraft  is  trimmed,  it  will  remain  in  level  flight  without  stick  inputs.  This  is  important  to  prevent 
hand  and  arm  fatigue  by  eliminating  the  need  for  a  constant  bias  on  the  stick.  The  aircraft  may 
need  to  be  re -trimmed  when  a  new  altitude  and  airspeed  operating  point  is  reached. 
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Weapon  Selection  -  The  choice  between  missiles  and  guns  can  be  toggled  using  the  thumb 
switch  on  the  stick. 

Fire  -  The  stick  trigger  is  used  to  fire  the  selected  weapon. 

Radio  Communications  -  The  pinkie  switch  on  the  stick  is  used  as  a  push-to-talk  switch  for 
radio  communications. 

Radar  Control  -  The  cursor  (top  center  on  the  back  of  the  throttle)  is  used  for  radar  functions. 
Pulling  the  cursor  up  cycles  the  designated  target  (if  there’s  more  than  one).  Moving  the  cursor 
to  the  right  toggles  the  radar  display  between  standby,  normal  mode,  and  HSI. 

Speed  Brake  -  The  hat  switch  on  the  side  of  the  throttle  opens  and  closes  the  speedbrake. 


Figure  4  -  SACS  Stick  Control  Functions 


Figure  5  -  SACS  Throttle  Functions 
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APPENDIX  E:  ACRONYM  LIST 


ACM 

Air  Combat  Maneuvering 

AAM 

Air-to-Air  Missile 

AMRAAM 

Advanced  Medium  Range  Air-to-Air  Missile 

AOA 

Angle  of  Attack 

DIS 

Distributed  Interactive  Simulation 

ECM 

Electronic  Countermeasures 

FPM 

Flight  Path  Marker 

HOTAS 

Hands-on  Throttle  and  Stick 

HSI 

Horizontal  Situation  Indicator 

HUD 

Heads-up  Display 

IAS 

Indicated  Airspeed 

kts 

Knots  (nautical  miles  per  hour). 

KIAS 

Knots  Indicated  Airspeed 

LCOS 

Lead  Computing  Optical  Sight 

RWR 

Radar  Warning  Receiver 

SACS 

SAFIRE  Air  Combat  Simulator 

TAA 

Target  Aspect  Angle 

TAS 

True  Airspeed 

VBMS 

Virtual  Battlespace  Management  System 
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